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IMAGE BLURRING 
PRIORITY 

This application claims the benefit of priority to United States provisional patent 
application no. 60/773,714, filed February 14, 2006, and United States patent application no. 
1 1/673,560, filed February 10, 2007. This application is related to United States patent 
application serial numbers 1 1/319,766, filed December 27, 2005 and 1 1/217,788, filed 
August 30, 2005, and 1 1/421,027, filed May 30, 2006. All of these applications are 
incorporated by reference. 

BACKGROUND 

1 . Field of the Invention 

The present invention relates to image blurring, and in particular, to a system and 
method for creating blur in an image to reduce the depth of field of the image. 

2. Description of the Related Art 

In digital cameras, the depth of field (hereinafter "DOF") is typically much greater 
than for cameras which use film due to the image sensor being somewhat smaller than in a 
35mm film negative. This means that portrait images captured with digital cameras, in 
particular, will tend to have the background in sharp focus, which is often not desirable as a 
photographer may wish to emphasize a person's face and de-emphasize the background of the 
picture. This problem may be corrected by careful photography combined with careful use of 
camera settings. 

Alternatively, portrait images may be blurred semi-manually by professional 
photographers using desktop computer image processing software after an image has been 
captured. This involves manual intervention and is often time-consuming. Nonetheless, such 
conventional blurring software may apply various techniques using convolution kernels to 
create blurring effects, as illustrated in Figures 1 and 2. 

Generically, convolution can be expressed according to the equation below: 

B = I*g 
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where B is the blurred image, / is the original image and g is the convolution kernel. 
Convolution blur may be applied on a pixel-by-pixel basis. So, for a particular pixel with 
coordinates (x,y), the convolution with a kernel of size (M x N) can be written as: 



The size and shape of the kernel influence the blurring result. The size determines the 
strength of the blur and therefore the perceived depth of the object. The shape determines the 
visual aspect of the blur and is related to what is called "circle of confusion". 
A circular kernel of a diameter D has the following analytical form 



and a geometrical shape of a cylinder or "pillbox", as is illustrated in Figure 1 . Referring 
now to Figures 2a-2b, the effects of a convolution kernel on a row of pixels within a flash 
image of a scene are illustrated. The most intense (bright areas) of the original image, i.e., in 
this example pixels 2 and 4 taken from left to right, undergo saturation clipping to the 
maximum of the dynamic intensity range (e.g., 255 for 8 bit / pixel image) as depicted by 
dashed outlines 20, while pixels 1, 3 and 5 are not clipped. Due to convolution, a resulting 
blurred image lacks the contrast and sharpness present in the scene, therefore creating a less 
appealing visual effect. Such blurring techniques simply do not achieve realistic results and 
do not resemble an image with a shallow DOF as desired. 



A method is provided for creating blur in an image acquired with a digital camera for 
printing or electronic display or both. A first image of a scene is acquired at a first exposure 
level, and a second image of nominally the same scene is acquired at a second exposure level. 
At least one region of the second image includes one or more pixels having saturated 
intensity values. For at least one of said saturated pixels, one or more values are extrapolated 
from the first image. A third image is generated including pixels of the second image having 
non-saturated values, and pixels having the one or more extrapolated values. At least a 
portion of the third image is blurred and re-scaled. 

The first image may be acquired without a flash and the second image may be 
acquired with a flash exposure level. 



j 




otherwise 



SUMMARY OF THE INVENTION 
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A value may be extrapolated for a saturated pixel by calculating at least one ratio of 
an intensity of a pixel in said first image to an intensity of at least one non-saturated pixel, 
and providing the extrapolated value for a selected pixel in accordance with the intensity 
ratio. A non-saturated pixel may include a pixel in the first image which corresponds to a 
5 pixel in the second image which has not undergone saturation clipping. 

One or more portions of said third image may be determined to correspond to 
foreground of the image, and portions may be determined which correspond to background of 
the image. The background portions may be blurred. 

Regions of said first and second images may be aligned at least in respect of the at 
1 0 least one portion. The first image may be a relatively low resolution image and the second 
image may be a higher resolution image, and the resolutions of the first and second images 
may be matched. The resolution the resolution of the second image may be downsampled to 
the resolution of said first image, or the resolution of the first image may be upsampled to the 
resolution of the 
1 5 second image, or a combination thereof. 

The blurring of the third image may be performed by applying a convolution kernel to 
the at least one portion. The kernel may be a circular kernel. 

The third image may include a copy of the second image modified by replacing the 
saturated intensity values with the extrapolated values. 
2 0 The third image may include the second image modified by replacing the saturated 

intensity values with the extrapolated values. 

The re-scaling may include scaling the blurred portion of the third image to a range 
associated with the second image. 

A further method is provided for generating a digital image. At least two images are 

2 5 acquired, including a full resolution image having one or more pixels with a saturated 

intensity value. One or more intensity ratios are determined from unsaturated pixels of 
another of the at least two acquired images. The full resolution image is modified including 
restoring the one or more intensity ratios determined from the unsaturated pixels of the other 
acquired image. A portion of the modified image is blurred, and the modified image is 

3 0 stored, displayed or printed, or combinations thereof. 

The modifying may include adding intensity to the one or more saturated pixels, or 
subtracting intensity from one or more unsaturated pixels, or a combination thereof. 
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The blurring may include determining one or more portions of the modified image 
which correspond to foreground of the image and portions which correspond to background 
of the image, and blurring the background portions. The blurred portion of the modified 
image may be rescaled. 

5 One or more processor readable storage devices are also provided with processor 

readable code embodied thereon for programming one or more processors to perform a 
method of creating blur in a digital image as disclosed herein. 

A digital image processing device is also provided which is operable to blur an 
acquired image, and includes a controller and the one or more processor readable storage 
1 0 devices. 

BRIEF DESCRIPTION OF THE DRAWING S 
Preferred and alternative embodiments will now be described, by way of example, 
with reference to the accompanying drawings, in which: 
1 5 Figure 1 depicts a conventional circular convolution kernel; 

Figures 2a-2b depict conventional image blurring using a circular convolution kernel; 
Figure 3 illustrates a flow diagram in accordance with one embodiment; 
Figures 4a-4c depict graphical representations of corresponding portions of a low- 
resolution or full resolution non- flash image and/or a preview image, a full resolution flash 

2 0 image or an image captured under another high light intensity exposure condition, and a high 

dynamic range (HDR) image derived therefrom according to another embodiment; and 

Figures 5a-5b depicts a graphical representation of a portion of a blurred image of the 
HDR image of Figure 4c, and a re-scaled blurred image derived therefrom according to 
another embodiment. 

25 

DETAILED DESCRIPTION OF THE EMBODIMENTS 
Techniques and components for obtaining a more realistic blur in a digitally-acquired 
image similar to the low DOF blur generated by a conventional film camera are provided. 

The low contrast and sharpness associated with conventional blurring techniques are 

3 0 mitigated or reduced or eliminated in accordance with embodiments described herein. 

Furthermore, lens effects of circle of confusion are emulated, as described in more detail 
below. 
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As illustrated in Figures 2a-2b, saturation clipping leads to a loss of high dynamic 
range, HDR, information about the image. In one embodiment of the present invention, this 
information is advantageously recovered to provide a more realistic blurring of an image of a 
scene. 

5 Referring now to Figure 3, final a low-resolution non- flash image of a scene (herein 

after referred to as a preview image) is captured at 100 prior to capturing a full-resolution 
flash image of a scene at 110. However, it will be appreciated that alternatively the preview 
may comprise any one or more of a series of captured non-flash low-resolution images. 
Alternatively, the non-flash image of the scene can be captured after the full-resolution flash 

1 0 version and as such may be a post- view image. The preview image may also be another full- 
resolution image or a down-sampled version thereof. For ease of reference, the term 
"preview image" is intended to include each of these. This embodiment may also be seen 
that the present invention can be applied even where the final full-resolution image 1 10 is 
captured without a flash. In such cases, the final full-resolution image 1 10 is may be taken at 

15 an exposure level which is the same as or different (usually higher) than, and is preferably 
higher than, that of the preview image 100 and so includes may include more pixels with 
saturated values than the preview image. 

The preview image may be utilized for various tasks such as auto-focus, white 
balance or to estimate exposure parameters. Preferably, the preview image is stored in a 

2 0 memory of the camera for use with other processing techniques. Alternatively the preview 
image may be stored in any suitable memory device, or may be only temporarily used or 
stored in the camera. 

The remainder of the blocks in the flow process illustrated at Figure 3 are described 
below with reference to Figures 4a-4c and 5a-5b, which further illustrate some of those 

2 5 process blocks. Referring to Figures 4a-4c, for purposes of illustration, intensity values of 

five pixels, which are preferably only a portion of one full row of multiple rows of pixels of a 
full resolution image or original image (Figure 4a), a preview image (Figure 4b) and a high 
dynamic range image (Figure 4c), or HDR image, are graphically represented. 

Intensity levels of the portion of the row of pixels of the full resolution image, that 

3 0 may have been taken with a flash or under another high light exposure condition, are 

graphically represented at Figure 4a. As earlier illustrated with reference to Figure 2a, this 
full resolution image has been subjected to saturation clipping due to the dynamic intensity 
range limit. In this case, pixels 2 and 4, taken from left to right, have been clipped from 
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values higher than 255 each to the limit of 255, while pixels 1, 3 and 5 have not been clipped. 
Therefore valuable information about the scene is eliminated from the full resolution, high 
exposure image. For one, the ratio of intensities of pixels 2 and 4 to pixels 1, 3 and 5 has 
been altered. The ratio between pixels 2 and 4 may also have been altered in the clipping 
5 process, and in fact was in the example illustrated at Figure 2a. 

As a preview image in the example illustrated at Figure 4b is preferably a non- flash 
image, saturation clipping generally does not occur and as such no exposure information or 
relatively little exposure information about the scene is lost. In Figure 4b, all five 
representative pixels has an intensity level below 255, so that none are clipped and the actual 

1 0 intensity ratios between each of the five pixels remains intact. 

In continuing with the description of the process of Figure 3, the preview and full 
resolution images are brought to the same resolution at 120, i.e., their resolutions are 
matched. In the preferred embodiment, the resolution of the images is matched by 
downsampling the flash image to the resolution of the preview image. Alternatively, the 

1 5 resolutions of the images may be matched by any combination of upsampling the preview 

image or downsampling the resolution of the flash image. However it will be appreciated any 
suitable means of bringing the images to the same resolution may be employed. Of course, if 
the resolutions are the same to begin with then 120 may be skipped. 

The preview and full resolution images are then aligned at 130, using image 

2 0 registration techniques, to compensate for any slight movement in the scene or camera 

between taking these images. Alignment may be performed globally across entire images or 
locally using various techniques, e.g., as may be described in co-pending US Patent 
Application No. 1 1/217,788 filed August 30, 2005, which is assigned to the same assignee, 
incorporated by reference, and not otherwise further expressly described herein. Again, if the 

2 5 images are already fully aligned, then the alignment at 130 would involve no modifications. 

Utilizing intensity information derived from the preview image, a high dynamic range 
(HDR) image is constructed at 140 from the full resolution image. The HDR image 
incorporates an estimate of the information (bright areas) eliminated from the flash image by 
saturation clipping. The dashed rectangles above each of the intensity level bar 

3 0 representations from each of the five pixels in Figure 4c illustrates these estimates. Figure 2a 

showed dashed rectangle only over pixels 2 and 4 which were the only pixels clipped. Figure 
4c has not only pixels 2 and 4 reduced to the limit of 255, but has pixels 1 , 3 and 5 reduced as 
well. One or both of pixels 2 and 4 could alternatively be reduced below 255 in the HDR 
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image of Figure 4c. The ratios of the intensities between the pixels 2 and 4 to pixels 1, 3 and 
5 in the HDR image of Figure 4c is preferably closer to actual object intensity ratios than 
Figure 4a. 

In one embodiment, the HDR image is achieved by determining an intensity ratio 
5 between two or more neighbouring pixels in the preview image, one of which will be clipped 
in the flash image; and the intensity values of one or more non-saturated pixels in the flash 
image. It will however be appreciated that the intensity ratio for each saturated pixel may be 
determined with respect to one or more non-neighbouring comparison pixels. Using this ratio 
information, the intensity of each of the clipped pixels of the flash image is extrapolated in 

1 0 proportion to the intensity ratio derived from the corresponding preview image pixel(s). 

For example, the ratio of the intensity of a first pixel of a preview image to the 
intensity of a neighbouring pixel of the preview image is determined. In the case where the 
first pixel's corresponding pixel in the flash image has been saturation clipped, the intensity 
of the clipped pixel is increased in accordance with the ratio information in order to restore 

1 5 the pixel to its original intensity ratio with respect to its neighbouring or comparison pixels. 
This process may be carried out for all saturated pixels to produce a HDR image. All of the 
pixels may be increased in intensity or decreased in intensity or a combination of increase 
and decrease, depending on other processes that may be in use such as selected fill-flash (see 
U.S. appln no. 10/608,810, incorporated by reference). The ratios may be adjusted precisely 

2 0 to preview image ratios, or otherwise as illustrated in Figure 4c, for example. In this way, the 
HDR image is made to resemble a flash image and/or high exposure image which was not 
subjected to saturation clipping. A portion of a row of pixels of the HDR image 
corresponding to pixels of the original and preview images is depicted graphically in Figure 
4c with the intensity ratios of the preview image being substantially the same as those 

2 5 provided in HDR image when the solid line and dashed line rectangles are each included. 

While this illustrative embodiment has been described in terms of providing a 
separate HDR image from the images 100, 110, another embodiment would provide for 
adjusting the values of the flash image 110 and using this adjusted image according to the 
below. In one embodiment, as disclosed in US Application No. 1 1/217,788 filed August 30, 

3 0 2005, the HDR image may undergo a digital segmentation process 135 to determine 

foreground and/or background within at least one portion of the image. In one exemplary 
implementation, the HDR image is compared to a preview non- flash image 100 of nominally 
the same scene. Overall light distribution may vary between the two images, because one 
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image or subset of images will be illuminated only with available ambient light while another 
is illuminated with direct flash light, thereby enabling the HDR image to be separated into 
foreground and background. As an alternative to using the HDR image, the full resolution 
and/or flash image 110 can be compared with a preview image 100 to perform 
5 foreground/background segmentation which could in turn be applied for use in processing the 
HDR image; or alternatively a flash and a non- flash preview image or one each full- 
resolution images could be used for foreground/background segmentation again for use in 
processing a HDR image, as could two flash or two non-flash images when captured at 
different exposure levels such that advantages illustrated at Figures 3-5 and in corresponding 

1 0 text description herein may be achieved. 

Alternatively, foreground and background regions of a HDR image may be separated 
at 135 (Figure 3) by a method disclosed in US provisional application serial number 
60/773,714 (Incorporated by reference). In this embodiment, one flash or non-flash image 
of a scene may be taken with the foreground more in focus than the background and which 

1 5 can be converted to a HDR image, e.g., according to the above. The HDR image may then be 
stored in, e.g., DCT-coded format or similar. A second out of focus image of nominally the 
same scene may be taken at 133 (Figure 3), and also stored in DCT-coded format. The two 
DCT-coded images may then be compared and regions of the HDR image assigned as 
foreground or background according to whether the sum of selected high order DCT 

2 0 coefficients are decreased or increased relative to equivalent regions of the second image. 

In one embodiment, as depicted in Figures 5a-5b, regions of the HDR image labeled 
as background from the above description may be blurred at 150 of Figure 3 with a circular 
kernel that resembles the PSF (point spread function) of a lens of a camera to emulate a real 
effect of optical blur. Figure 5a illustrates intensity levels of five exemplary pixels of the 

2 5 HDR image which are blurred. A circular shaped kernel may be advantageously employed 
because it approximates a real lens aperture effect. Also, since the lens does not amplify or 
reduce the amount of light passing through, the convolution kernel is derived such as the sum 
of all its values equals 1, i.e.: 

M,N 

Uj 

30 

Other suitably shaped kernels may be utilized. The range of the blurred image 
produced in step 150 of Figure 3 is then scaled back, as illustrated at Figure 5b and at block 
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160 of Figure 3, to the range of the full resolution image to produce a realistically blurred 
image at block 170 similar to the low depth-of-field blur generated by a film-based camera. 

It will be seen that many variations of the above embodiments are possible. For 
example, image processing software described in Figure 3 can be implemented completely in 
5 a camera or as part of an external processing device such as a desktop computer which is 
provided with the images 100, 110, 133. 

While an exemplary drawings and specific embodiments of the present invention have 
been described and illustrated, it is to be understood that that the scope of the present 
invention is not to be limited to the particular embodiments discussed. Thus, the 
1 0 embodiments shall be regarded as illustrative rather than restrictive, and it should be 

understood that variations may be made in those embodiments by workers skilled in the arts 
without departing from the scope of the present invention as set forth in the claims that follow 
and their structural and functional equivalents. 

In addition, in methods that may be performed according to the claims below and/or 
1 5 preferred embodiments herein, the operations have been described in selected typographical 
sequences. However, the sequences have been selected and so ordered for typographical 
convenience and are not intended to imply any particular order for performing the operations, 
unless a particular ordering is expressly provided or understood by those skilled in the art as 
being necessary. 

2 0 In addition, that which is described as background, the invention summary, the 

abstract, the brief description of the drawings and the drawings themselves, as well as all 
references cited above herein, and US published applications nos. 2005/0068448, 
2006/0098237, 2006/0204110, 2006/0098890, 2006/0288071, and 2006/0285754, and US 
provisional applications nos. 60/746,363 and 60/803,980, which are assigned to the same 

2 5 assignee, are hereby incorporated by reference into the detailed description of the preferred 
embodiments as providing alternative embodiments. 
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Claims: 

Claim 1 : A digital image processing device operable to blur an acquired image, comprising a 
controller and one or more processor readable storage devices having processor readable code 
5 embodied thereon, said processor readable code for programming one or more processors to 
perform a method of creating blur in a digital image, the method comprising: 

(a) acquiring a first image of a scene taken at a first exposure level; 

(b) acquiring a second image of nominally the same scene taken at a second exposure 
level, wherein at least one region of said second image includes one or more pixels having 

1 0 saturated intensity values; 

(c) for at least one of said saturated pixels, extrapolating one or more values from said 
first image; 

(d) generating a third image comprising pixels of said second image having non- 
saturated values, and pixels having said one or more extrapolated values; 

1 5 (e) blurring at least a portion of the third image; and 

(f) re-scaling the blurred portion of the third image. 

Claim 2 : The digital image processing device of claim 1 , wherein the first image is acquired 
without a flash and wherein the second image is acquired with a flash exposure level. 

20 

Claim 3 : The digital image processing device of claim 1 , wherein extrapolating a value for a 
saturated pixel comprises: 

calculating at least one ratio of an intensity of a pixel in said first image to an intensity 
of at least one non-saturated pixel; and 

2 5 providing said extrapolated value for a selected pixel in accordance with said intensity 

ratio. 

Claim 4: The digital image processing device of claim 3, wherein said non-saturated pixel 
comprises a pixel in said first image which corresponds to a pixel in said second image which 

3 0 has not undergone saturation clipping. 

Claim 5: The digital image processing device of claim 1, wherein the method further 
comprises: 
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determining one or more portions of said third image which correspond to foreground 
of the image and portions which correspond to background of the image; and 
blurring said background portions. 

5 Claim 6: The digital image processing device of claim 1, wherein said blurring of said third 
image comprises applying at least one convolution kernel to said at least one portion. 

Claim 7: The digital image processing device of claim 6, wherein said kernel comprises a 
circular kernel. 

10 

Claim 8 : The digital image processing device of claim 1 , wherein said third image comprises 
a copy of said second image modified by replacing the saturated intensity values with the 
extrapolated values. 

1 5 Claim 9: The digital image processing device of claim 1, wherein said third image comprises 
said second image modified by replacing the saturated intensity values with the extrapolated 
values. 

Claim 10: The digital image processing device of claim 1, wherein said re-scaling comprises 
2 0 scaling said blurred portion of said third image to a range associated with said second image. 

Claim 1 1 : A digital image processing device operable to blur an acquired image, comprising 
a controller and one or more processor readable storage devices having processor readable 
code embodied thereon, said processor readable code for programming one or more 

2 5 processors to perform a method of generating a digital image, the method comprising: 

(a) acquiring at least two images, including a full resolution image having one or 
more pixels with a saturated intensity value; 

(b) determining one or more intensity ratios from unsaturated pixels of another of the 
at least two acquired images; 

3 0 (c) modifying the full resolution image including restoring the one or more intensity 

ratios determined from the unsaturated pixels of said other acquired image; 
(d) blurring a portion of the modified image; and 
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(e) printing, image processing, or electronically storing, transmitting or displaying the 
modified image, or combinations thereof. 

Claim 12: The device of claim 11, wherein the modifying comprises adding intensity to the 
5 one or more saturated pixels. 

Claim 13: The device of claim 12, wherein the modifying further comprises subtracting 
intensity from one or more unsaturated pixels. 

1 0 Claim 14: The device of claim 11, wherein the modifying further comprises subtracting 
intensity from one or more unsaturated pixels. 

Claim 15: The device of claim 11, the blurring comprising: 

determining one or more portions of said modified image which correspond to 
1 5 foreground of the image and portions which correspond to background of the image; and 
blurring said background portions. 

Claim 16: The device of claim 11, the method further comprising re-scaling the blurred 
portion of the modified image. 

20 

Claim 17: A method for creating blur in an image acquired with a digital camera for printing, 
image processing or analysis, or electronic storage, transmission or display, or combinations 
thereof, said method comprising: 

(a) acquiring a first image of a scene taken at a first exposure level; 

2 5 (b) acquiring a second image of nominally the same scene taken at a second exposure 

level, wherein at least one region of said second image includes one or more pixels having 
saturated intensity values; 

(c) for at least one of said saturated pixels, extrapolating one or more values from said 
first image; 

3 0 (d) generating a third image comprising pixels of said second image having non- 

saturated values, and pixels having said one or more extrapolated values; 

(e) blurring at least a portion of the third image; and 

(f) re-scaling the blurred portion of the third image. 
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Claim 18: The method of claim 17, further comprising: 

determining one or more portions of said third image which correspond to foreground 
of the image and portions which correspond to background of the image; and 

blurring said background portions. 

Claim 19: The method of claim 17, wherein said blurring of said third image is performed by 
applying at least one convolution kernel to said at least one portion. 

Claim 20: The method of claim 17, wherein said third image comprises said second image 
modified by replacing the saturated intensity values with the extrapolated values. 
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